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PRELIMINARY AMENDMENT 



Assistant Commissioner for Patent 
Washington, D.C. 20231 

Dear Sir: 

Please amend the application, without prejudice, as follows: 
In the Specification 

Please change the last paragraph of page 15 which continues to page 16 to read as: 

Referring again to Fig. 1, when a patient's colon is to be rendered and a contrast 
agent has been used to opacify stool and/or residual water, the segmentation step 35 
includes the step of digitally removing the opacified stool/water from the CT image data in 
order to render an unimpeded colon. Digital removal of opacified feces is difficult due to 
the fact that simple subtraction leaves behind edge gradients. The appearance of edge 
gradients is illustrated in Figs. 15-17. Fig. 15 is a schematic representation of a section of 
image data representing hypothetical CT image data showing the measured X-ray 
attenuation values associated with a two dimensional array of voxels. Soft tissue is 
represented by a value of 100, opacified stool by a value of 200, and air by a value of 0. In 
this simplified model, actual interfaces between different species is presumed to lie on the 
boundary between adjacent voxels. Accordingly, an interface 601 between soft tissue and 



opacified stool appears as the boundary between the voxels having a value of 100 and those 
having a value of 200. Similarly, an interface 600 between opacified stool and air appears 
as the boundary between the voxels having a value of 200 and those having a value of 0. 

Please change the first paragraph of the Application, which was added by the first 
preliminary amendment, to read as: 

This application claims the benefit of co-pending U.S. Provisional Application 
Serial No. 60/109,966, filed November 25, 1998, such application being incorporated 
herein by reference. This application also claims the benefit of co-pending U.S. 
Application Serial No. 09/299,061 filed on April 23, 1999, which is a continuation of U.S. 
Application Serial No. 08/805,584 filed on February 24, 1997, now issued as U.S. Patent 
5,920,319. 



In the Claims: 

Please add new claims: 

52. A method for guiding biopsy of a tissue by accessing the tissue via a 
three-dimensional structure having a lumen comprising the steps of: 

a. providing a three-dimensional volume of data representing at least one 
physical property associated with the three-dimensional structure; 

b. isolating a first region of interest of the structure having the lumen; 

c. rendering the first region of interest in an interactive three-dimensional 
display; 

d. inserting a simulated biopsy needle within the lumen of the rendered first 
region of interest; 

e. determining the position of the simulated biopsy needle within the rendered 
first region of interest having the lumen; and 

f . displaying the simulated biopsy needle within the rendered first region of 
interest having the lumen. 



53. 



The method of claim 52 comprising the step of isolating the first region of interest 
from the volume of data based on a selected value of the physical property associated 



with the three-dimensional structure. 



54. The method of claim 53 wherein the step of isolating the first region of interest 
comprises segmenting the first region of interest. 

55. The method of any one of claims 52 - 54 comprising: 

providing a three-dimensional volume of data representing at least one 
physical property associated with the tissue; 

isolating a second region of interest of the structure having the tissue; 

rendering the second region of interest in the interactive three-dimensional 
display; and 

creating at least one target zone of needle placement at a selected site of the 
rendered tissue. 

56. The method of claim 55 comprising the step of isolating the second region of interest 
from the volume of data based on a selected value of the physical property associated 
with the tissue. 

57. The method of claim 56 wherein the step of isolating the second region of interest 
comprises segmenting the second region of interest. 

58. The method of claim 55 wherein the position determining step includes: 

determining the position of the needle relative to a selected target zone; 

generating a ready-to-strike indicator if the needle is within a desired location 
relative to the selected target zone; and 

calculating a movement pathway for the needle to strike the target zone if the 
needle is not within a desired location relative to the selected target zone. 

59. The method of claim 52 comprising: 

providing a three-dimensional volume of data representing at least one 
physical property associated with mediastinal tissue; 

isolating a second region of interest of the structure having the mediastinal 

tissue; 

rendering the second region of interest in the interactive three-dimensional 
display; and 

positioning the needle so as to avoid the rendered mediastinal tissue. 

60. A system for interactively displaying three dimensional structures comprising: 

a. volume formation means for forming a three-dimensional volume of data 
representing at least one physical property associated with a three dimensional 
structure; 

b. a wireframe generator for generating a wireframe model having vertices of a 
region of interest representing the three-dimensional structure; 

c. refinement means for refining the wireframe model by using a geometric 
deformable model; and 

d. a renderer for rendering the wireframe model in an interactive 
three-dimensional display. 
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61. The system of claim 60 wherein the renderer produces a virtual three-dimensional 
environment. 

62. The system of claim 60 comprising an isolator for isolating the region of interest from 
the volume of data based on selected values of the physical property representing the 
region of interest. 

63. The system of claim 62 wherein the renderer produces a virtual three-dimensional 
environment. 

64. The system of claim 62 wherein the isolator comprises a surface model identification 
means for identifying a surface model and wherein the wireframe generator is adapted 
to create a wireframe model of the surface model. 

65. The system of claim 62 wherein the isolator comprises a segmentation producer for 
segmenting the region of interest from the volume of data. 

66. The system of claim 65 wherein the isolator comprises an isosurface creation means 
for creating an isosurface of the isolated region of interest. 

67. The system of claim 66 wherein the wireframe generator is adapted to create a 
wireframe model of the isosurface. 

68. The system of any one of claims 60, 64, or 66 comprising: 

a vertex grouper for grouping the vertices of the wireframe model into regions 
having abnormal structure; and 

an analyzer for analyzing the regions having abnormal structure. 

69. The system of claim 68 wherein the vertex grouper comprises: 

a mean curvature calculator for calculating a mean curvature at each of a 
selected number of vertices; 

a gaussian curvature calculator for calculating a gaussian curvature at each of 
the selected number of vertices; and 

classification means for classifying the selected vertices according to the mean 
curvatures and the gaussian curvatures of the vertices. 

70. A computer-implemented method for interactively displaying a three-dimensional 
rendering of a structure having a lumen comprising the steps of: 

a. providing a three-dimensional volume of data representing at least one 
physical property associated with the structure having the lumen; 

b. generating a wireframe model having vertices representing a region of interest 
of the structure having the lumen; 

c. refining the wireframe model using a geometric deformable model; and 

d. rendering the wireframe model in an interactive three-dimensional display. 

71. The method of claim 70 wherein the wireframe model comprises a plurality of 
vertices, the method comprising: 
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grouping the vertices of the wireframe model into regions having a 
characteristic indicating abnormal structure; 

analyzing the regions having abnormal structure to identify regions having a 
high degree of abnormality; and 

rendering the wireframe model in an interactive three-dimensional display to 
indicate the regions having abnormal structure. 

72. The method of claim 71 wherein the grouping step comprises: 

calculating a mean curvature at selected vertices; 
calculating a gaussian curvature at the selected vertices; 
classifying the selected vertices according to the mean curvature and the 
gaussian curvature; and 

grouping adjoining selected vertices into patches of same vertex classification. 

73. The method of claim 72 wherein the analyzing step comprises labeling patches having 
an elliptical classification as a potential lesion. 

74. The method of claim 73 wherein the analyzing step comprises eliminating from the 
grouping selected elliptical patches which point outwardly from the lumen. 

75. The method of claim 74 wherein the analyzing step comprises calculating principal 
curvatures of the patches from the mean and gaussian curvatures and eliminating from 
the grouping any elliptical patches having substantially different principal curvatures. 

76. The method of claim 70 comprising calculating a normal vector for each vertex in the 
wireframe model. 

77. The method of claim 76 wherein the refinement step comprises moving each vertex of 
the wireframe model along its normal vector direction at a speed that is a function of a 
surface curvature and a doubt level of a boundary estimate at that point. 

78. The method of claim 76 wherein the refinement step uses a three-dimensional region- 
growing model as a starting point for the geometric deformable model. 

79. The method of claim 78 wherein the refinement step uses a three-dimensional region- 
growing model as a starting point for the geometric deformable model. 

80. The method of claim 70 wherein the step of generating a wireframe model comprises 
the steps of identifying a surface model of the region of interest and generating a 
wireframe model of the surface model. 

81. The method of claim 80 comprising: 

grouping the vertices of the wireframe model into regions having a 
characteristic indicating abnormal structure; 

analyzing the regions having abnormal structure to identify regions having a 
high degree of abnormality; and 

rendering the wireframe model in an interactive three-dimensional display to 
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indicate the regions having abnormal structure. 

82. The method of claim 81 wherein the grouping step comprises: 

calculating a mean curvature at selected vertices; 
calculating a gaussian curvature at the selected vertices; 
classifying the selected vertices according to the mean curvature and the 
gaussian curvature; and 

grouping adjoining selected vertices into patches of same vertex classification. 

83. The method of claim 82 wherein the analyzing step comprises labeling patches having 
an elliptical classification as a potential lesion. 

84. The method of claim 83 wherein the analyzing step comprises eliminating from the 
grouping selected elliptical patches which point outwardly from the lumen. 

85. The method of claim 84 wherein the analyzing step comprises calculating principal 
curvatures of the patches from the mean and gaussian curvatures and eliminating from 
the grouping any elliptical patches having substantially different principal curvatures. 

86. The method of claim 70 comprising isolating the region of interest from the volume of 
data based on a selected value of the physical property representing the region of 
interest. 

87. The method of claim 86 wherein the step of isolating comprises segmenting the region 
of interest. 

88. The method of claim 87 comprising creating an isosurface of the region of interest. 

89. The method of claim 88 wherein the step of generating a wireframe model comprises 
generating a wireframe model of the isosurface. 

90. The method of claim 89 wherein, the wireframe model comprises a plurality of 
vertices, the method comprising: 

grouping the vertices of the wireframe model into regions having a 
characteristic indicating abnormal structure; 

analyzing the regions having abnormal structure to identify regions having a 
high degree of abnormality; and 

rendering the wireframe model in an interactive three-dimensional display to 
indicate the regions having abnormal structure. 

91. The method of claim 90 wherein the grouping step comprises: 

calculating a mean curvature at selected vertices; 
calculating a gaussian curvature at the selected vertices; 
classifying the selected vertices according to the mean curvature and the 
gaussian curvature; and 

grouping adjoining selected vertices into patches of same vertex classification. 
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92. The method of claim 91 wherein the analyzing step comprises labeling patches 
having an elliptical classification as a potentialTesion. 

93. The method of claim 92 wherein the analyzing step comprises eliminating from the 
grouping selected elliptical patches which point outwardly from the lumen. 

94. The method of claim 93 wherein the analyzing step comprises calculating principal 
curvatures of the patches from the mean and gaussian curvatures and eliminating from 
the grouping any elliptical patches having substantially different principal curvatures. 

95. A computer-implemented method for interactively displaying a three-dimensional 
rendering of a structure having a lumen comprising the steps of: 

a. providing a three-dimensional volume of data representing at least one 
physical property associated with the structure having the lumen; 

b. generating a wireframe model having a plurality of vertices to represent the 
structure having the lumen; 

c. grouping the vertices by: 

calculating a mean curvature at each of a selected number of vertices, 
calculating a gaussian curvature at each of the selected number of 
vertices, 

classifying the selected vertices according to the mean curvatures and 
the gaussian curvatures of the vertices, and 

collecting adjoining vertices of similar classification into patches of 
similar vertex classification; and 

d. rendering the wireframe model in an interactive three-dimensional display. 

96. The method of claim 95 comprising isolating a region of interest from the volume of 
data based on a selected value of the physical property representing the region of 
interest. 

97. The method of claim 96 wherein the step of isolating comprises segmenting the region 
of interest. 

98. The method of claim 95 wherein the step of generating a wireframe model comprises 
the steps of identifying a surface model of the region of interest and generating a 
wireframe model of the surface model. 

99. The method of claim 95 comprising creating an isosurf ace of the region of interest. 

100. The method of claim 95 comprising labeling patches having an elliptical classification 
as a potential lesion. 

101. The method of claim 100 comprising eliminating from the grouping selected elliptical 
patches which point outwardly from the lumen. 

102. The method of claim 101 comprising calculating principal curvatures of the patches 
from the mean and gaussian curvatures and eliminating from the grouping any 
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elliptical patches having substantially different principal curvatures. 

103. A system for interactively displaying three dimensional structures comprising: 

a. volume formation means for forming a three-dimensional volume of data 
representing at least one physical property associated with a three dimensional 
structure; 

b. a wireframe generator for generating a wireframe model having a plurality of 
vertices to represent a region of interest of the three-dimensional structure; 

c. a vertex grouper for collecting adjoining vertices having similar classifications 
based on mean and gaussian curvatures into patches of similar vertex 
classification; and 

d. a renderer for rendering the wireframe model in an interactive 
three-dimensional display. 

104. The system of claim 103 wherein the renderer produces a virtual three-dimensional 
environment. 



105. The system of claim 103 comprising an isolator for isolating the region of interest 
from the volume of data based on selected values of the physical property representing 
the region of interest. 

106. The system of claim 105 wherein the renderer produces a virtual three-dimensional 
environment. 

107. The system of claims 105 or 106 wherein the isolator comprises a segmentation 
producer for segmenting the region of interest from the volume of data. 

108. The system of claim 105 wherein the isolator comprises a surface model identification 
means for identifying a surface model and wherein the wireframe generator is adapted 
to create a wireframe model of the surface model. 

109. The system of claim 105 wherein the isolator comprises an isosurface creation means 
for creating an isosurface of the isolated region of interest. 

110. The system of claim 103 wherein the vertex grouper comprises: 

a mean curvature calculator for calculating a mean curvature at each of a 
selected number of vertices; 

a gaussian curvature calculator for calculating a gaussian curvature at each of 
the selected number of vertices; and 

classification means for classifying the selected vertices according to the mean 
curvatures and the gaussian curvatures of the vertices. 
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REMARKS 

Applicants have added new claims 52-110 to claim additional features of Applicants' 
invention. Applicants have also amended the specification at page 15 to make the recitation 
of reference numerals 600 and 601 consistent with the usage of such reference numerals in 
Figure 15. No new matter has been added. Applicants have added a claim to additional 
related applications. 



Respectfully submitted, 




Donald R. Piper, 
PTO Reg. No. 29,337 

DANN DORPMAN HERRELL & SKILLMAN, PC 
1601 Market Street, Suite 720 
Philadelphia, PA 19103 
Phone: (215) 563-4100 
Fax: (215) 563-4044 



ATTACHMENT 

The following is a marked up version of the first" paragraph of the Application: 

This application claims the benefit of U.S. Provisional Application Serial No. 
60/109,966, filed November 25, 1998, such application being incorporated herein by 
reference. This application also cl aims the benefit of co-pending U.S. Application Serial 
No. 09/299,061 filed on April 23. 1 999. which is a continuation of U.S. Application Serial 
No. 08/805,584 filed on February 24. 1997. now issued as U.S. Patent 5.920.319. 

The following is a marked up version of the last paragraph of page 15 which continues to 
page 16: 

Referring again to Fig. 1, when a patient's colon is to be rendered and a contrast 
agent has been used to opacify stool and/or residual water, the segmentation step 35 
includes the step of digitally removing the opacified stool/ water from the CT image data in 
order to render an unimpeded colon. Digital removal of opacified feces is difficult due to 
the fact that simple subtraction leaves behind edge gradients. The appearance of edge 
gradients is illustrated in Figs. 15-17. Fig. 15 is a schematic representation of a section of 
image data representing hypothetical CT image data showing the measured X-ray 
attenuation values associated with a two dimensional array of voxels. Soft tissue is 
represented by a value of 100, opacified stool by a value of 200, and air by a value of 0. In 
this simplified model, actual interfaces between different species is presumed to lie on the 
boundary between adjacent voxels. Accordingly, an interface [600] 601 between soft tissue 
and opacified stool appears as the boundary between the voxels having a value of 100 and 
those having a value of 200. Similarly, an interface [601] 600 between opacified stool and 
air appears as the boundary between the voxels having a value of 200 and those having a 
value of 0. 
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IDENTIFICATION OF THE INTERNATIONAL APPLICATION 


INTERNATIONAL APPLICATION NO. 
PCT/US99/28030 


INTERNATIONAL FILING DATE 
24 NOV 99 



O PRELIMINARY AMENDMENT 

Iff Please amend the specification of the above identified application 
Ms follows : 

iy Before the Field of the Invention, insert the following: 
QH- RELATED APPLICATIONS 

Jjt This application claims the benefit of U.S. Provisional Application 
gprial No. 60/109,966, filed November 25, 1998, such application being 
Hicorporated herein by reference. -- 

REMARKS 

The application has been amended to contain a specific reference to 
an earlier filed application. No new matter has been added. 

Re spec t f u 1 ly submi 1 1 ed , 

DANN, DORFMAN, HERRELL AND SKILLMAN 
A Professional Corporation 
Attorney for Applicant 




DONALD R. PIPER, JR. 

PTO Registration No. 29,337 



Telephone: 215/563-4100 
Facsimile: 215/5 63-4044 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re the Application of 
VINING et al. 

.Application No. 09/856,683 

Filed: May 23, 2001 

For: Virtual Endoscopy with 
Improved Image Segmentation 
and Lesion Detection 



Examiner : 
Group Art Unit: 



THIRD PRELIMINARY AMENDMENT 



Please change the Related Applications section 
of the above-identified application to read as follows: 

This application claims the benefit of U.S. 
Provisional Application Serial No. 60/109,966, filed 
November 25, 1998, such application being incorporated 
herein by reference. This application also claims the 
benefit of and is a continuation-in-part of co-pending 
U.S. Application Serial No. 09/299,061 filed on April 23, 
1999, which is a continuation of U.S. Application Serial 
No. 08/805,584 filed on February 24, 1997, now issued as 
U.S. Patent 5,920,319. 



Remarks 

The Related Application section of the 
specification has been amended contain editorial 
revisions. No new matter has been added. 

Respectfully submitted, 



DANN, DORFMAN, HERRELL AND SKILLMAN 
A Professional Corporation 
Attorney for Applicant (s) 




DONALD R. PIPER, JR., ESQ. 
PTO Registration No. 29,337 



Telephone: 215/563-4100 
Facsimile: 215/563-4044 



ATTACHMENT 



The following is a marked up version of the Related 
Application section of the specification: 

This application claims the benefit of [co- 
pending] U.S. Provisional Application Serial No. 
60/109,966, filed November 25, 1998, such application 
being incorporated herein by reference. This application 
also claims the benefit of and is a continuation-in-part 
of co-pending U.S. Application Serial No. 09/299,061 filed 
on April 23, 1999, which is a continuation of U.S. 
Application Serial No. 08/805,584 filed on February 24, 
1997, now issued as U.S. Patent 5,920,319. 
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FIELD OF THE INVENTION 

The present invention relates to a system and 
method for interactively displaying a three- 
dimensional rendering of a structure and, more 
particularly, to a system and method for automatically 
analyzing such structures for potential abnormalities. 

BACKGROUND OF THE INVENTION 

For many forms of cancer, early detection is 
essential for a favorable prognosis. A cancerous 
growth must be detected at an early stage before the 
cancer is allowed to grow and spread. This is 
particularly true for colorectal and lung cancers. As 
a result, endoscopic techniques have been developed to 
examine the colon and tracheobronchial airways for the 
growth of precancerous and cancerous masses. 

Regarding colorectal cancer, the American Cancer 
Society and the National Cancer Institute recommend 
routine screening in order to provide a means for 
early detection of abnormal growths. Under those 
guidelines over 50 million Americans should undergo 
annual colorectal screening. However, only a small 
percentage of the population undergoes colonoscopy 



screening each year. The underutilization of such 
screening exists, at least in part, because 
conventional colonoscopies require a patient to be 
sedated so that a flexible endoscope can be inserted 
into the patient's anus. Another problem is that 
conventional colonoscopies fail to provide access to 
the entire colon in approximately 15% of cases. 
Colonoscopies also expose patients to the risk of 
bowel perforation. Accordingly, it is understandable 
that patients are reluctant to undergo, or repeatedly 
subject themselves to, such an invasive procedure. 

Consequently, virtual endoscopic techniques have 
been, and are continuing to be developed. Virtual 
endoscopy that utilizes computer reformation of 
radiologic cross-sectional images is a minimally 
invasive alternative to conventional fiberoptic 
endoscopy. Virtual endoscopy reduces the risk of 
perforation, does not require any sedation, and is 
considerably less expensive than the fiberoptic 
endoscopy method. For example, virtual colonoscopy 
techniques generally require bowel cleansing, gas 
distension of the colon, a 10-60 second helical 
computed tomography (CT) scan of a patient's abdomen 
and pelvis, and human visual analysis of multi-planar 
two-dimensional (2D) and three-dimensional (3D) images 
created from CT data. 

Although virtual colonoscopy techniques provide 
excellent images of the colon in three -dimensions, a 
correct diagnosis relies upon a physician's ability to 
properly identify small (approximately 3-10 mm), and 
sometimes subtle, masses within hundreds of 
multiplanar two-dimensional and three-dimensional 
images. Such human inspection is time consuming, 



10 



15 



WO 00/32106 PCT/US99/28030 

tedious, expensive, and under certain circumstances 
prone to error of interpretation. Accordingly, it 
would be highly beneficial to provide an automatic 
virtual endoscopic system and method for automatically 
analyzing and/or detecting abnormalities, such as 
abnormal growths, in a targeted structure or organ 
system, such as the walls of a colon or trachea. 

SUMMARY OF THE INVENTION 

In accordance with the present invention, a 
computer- implemented method and computer system are 
provided for interactively displaying a three- 
dimensional rendering of a structure, generally having 
a lumen. In a specific application, the method may be 
utilized for analyzing regions having certain 
characteristics of wall structure, such as thickness 
or shape, to detect, for example, abnormal masses. In 
accordance with the method of the present invention, a 
three-dimensional volume of data is formed from a 
series of two-dimensional images representing at least 
one physical property associated with the three- 
dimensional object. A surface of a selected region of 
interest, optionally an isosurface, is created in the 
computer from the volume of data based on selected 
25 values of the physical properties representing the 

selected region of interest, A wireframe model of the 
surface is generated by the computer. The wireframe 
model may be purposefully deformed by the computer to 
more accurately match the object of interest. The 
wireframe model is analyzed to detect, by computer, 
those sections of the object having the selected 
characteristic such as abnormal wall structure. For 
example, the wireframe model includes a plurality of 
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vertices. The computer- implemented method groups the 
vertices of the wireframe model into populations 
having a characteristic indicating abnormal wall 
structure. The wireframe model with highlighted 
abnormal portions is then rendered by the computer 
into an interactive three-dimensional display on the 
computer monitor. Alternatively, the locations of 
selected abnormal portions may be stored and indexed 
in the computer without necessity of an interactive 
display. 

In specific applications, a system for 
interactively displaying three-dimensional structures 
is provided. Volume formation means is included for 
forming a three-dimensional volume of data from a 
series of two-dimensional images representing at least 
one physical property associated with a three- 
dimensional body. A wireframe modeling means is 
provided for producing a wireframe model of a selected 
region or object of interest. A wireframe deforming 
means is provided for deforming the wireframe model to 
more accurately represent the object of interest. 
Finally, rendering means may optionally be provided 
for rendering the wireframe model in an interactive 
three-dimensional display, therein producing a virtual 
three-dimensional environment . 

In a specific embodiment, segmentation means may 
be provided for segmenting the object of interest from 
the volume of data based on selected values of the 
physical property representing the region of interest. 
In addition, the modeling means may function to 
produce the wireframe model from the segmented region 
of interest. 



WO 00/32106 PCT/US99/28030 

The wireframe model may be produced to include a 
plurality of vertices, with each vertex having a 
coordinate position. In such an embodiment, the 
deforming means may include refinement means for 
refining the wireframe model by adjusting the 
coordinate positions of the vertices to more 
accurately represent the object of interest. 

As a further option, the vertices of the 
wireframe model may be grouped into regions having a 
characteristic indicating abnormal structure. The 
regions of abnormal structure may then be analyzed to 
identify regions having a high degree of abnormality. 
The abnormal structure can be designated and indexed 
by the computer either as a comprehensive listing or 
as a listing dependent on the degree of abnormality. 

The system may also include provision for 
targeting abnormalities and for determining the 
position of a biopsy needle within the three- 
dimensional virtual environment. In such an 
embodiment, an interactive three-dimensional display 
of a rendered wireframe model of a selected object of 
interest is displayed from the perspective indicating 
the position of the biopsy needle within the three- 
dimensional virtual environment. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing summary, as well as the following 
detailed description of the preferred embodiments of 
the present invention, will be better understood when 
read in conjunction with the accompanying drawings, in 
which : 

Fig. 1 is a flowchart representing a method in 
accordance with the present invention of producing an 
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interactive, three-dimensional rendering of a selected 
organ system, such as an organ containing a lumen, and 
of detecting potentially abnormal areas of the organ 
system; 

Fig. 2 is a flowchart representing a method for 
analyzing the shape of a population representing a 
potentially abnormal area within the selected organ 
system; 

Figs . 3 a~b are graphical representations of a 
method for treating evolving surfaces as the level 
surfaces of an evolving scalar function; 

Fig. 4 is a block diagram of a system used in the 
method of the present invention; 

Fig. 5 is a schematic representation of a two- 
dimensional plane through a wireframe model of an area 
of abnormal wall structure illustrating a method for 
determining the convexity of the area; 

Figs, 6 a-d are diagrams illustrating methods of 
calculating wall thickness at a vertex of a wireframe 
model by analyzing the intensity profile along the 
normal vector originated at the vertex; 

Fig. 7 is a schematic view of a display window 
for displaying a volume rendered image, a surface 
rendered image, and three multiplanar images along 
three different planes of a selected organ system; 

Fig. 8 is a schematic diagram depicting the 
cross -sectional area of a lumen defined by a plane 
through the lumen; 

Figs. 9 a-d are diagrams representing methodology 
for calculating the cross sectional area of the lumen 
defined by the cutting plane shown in Fig. 8; 

Fig. 10 is a schematic view of a display window 
for displaying a volume rendered image, a surface 
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rendered image, and three multiplanar images along 
three different planes of a selected organ system 
combined in a single window display; 

Fig. 11 is a block diagram of an alternate 
embodiment of the system used in the method of the 
present invention; 

Fig, 12 is a flowchart representing a method for 
creating a selected surface of a region of interest; 

Fig. 13 is a flowchart representing an alternate 
method for analyzing the shape of a population; 

Fig. 14 is a flowchart representing a method for 
guiding endoscopic biopsy procedures in accordance 
with the present invention; 

Fig. 15 is a schematic representation of section 
of two-dimensional image data containing hypothetical 
soft tissue/opacif ied stool and opacified stool/air 
interfaces ; 

Fig. 16 is a schematic representation of a 
simplified section of two-dimensional image data 
containing soft tissue/opacif ied stool and opacified 
stool/air interfaces; and 

Fig. 17 is a schematic representation of the 
section of two-dimensional image data of Fig. 16 
showing edge gradient effects resulting from a simple 
digital subtraction scheme. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention generally relates to a 
computer- implemented method and system, as 
schematically represented in Figs. 1, 2, and 4 for 
generating interactive, three-dimensional renderings 
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of three-dimensional structures generally having a 
lumen. The structures are usually in the general form 
of selected regions of a body and in particular, human 
or animal body organs which have hollow lumens such as 
colons, blood vessels, and airways. In accordance 
with the present invention, the interactive three- 
dimensional renderings are generated in a computer- 
controlled process from a series of two-dimensional, 
cross-sectional images of the selected body organ 
acquired, for example, from a helical computed 
tomography (CT) scan. The three-dimensional 
renderings are interactive in that such renderings can 
be manipulated by a user on a visual display of a 
computer system, such as a computer monitor, to enable 
movement in, around and through the three-dimensional 
structure, or slices or cuts of such three-dimensional 
structure, while simultaneously displaying multiplanar 
views centered on a point that changes in response to 
the movement, or on a point selected by the user. 

The computer-generated interactive three- 
dimensional renderings can be constructed from surface 
renderings of the organ system or from volume 
renderings of the organ system and surrounding anatomy 
and may be presented from surface and volume 
renderings simultaneously in the same three- 
dimensional display window. In a selected embodiment, 
a surface rendering of an organ can be incorporated 
into a volume rendering of the surrounding anatomy 
using texture memory in order to achieve a combined 
surface/volume rendering, or a hybrid rendering 
effect, wherein the surface rendered organ may be 
highlighted in the context of the volume -rendered 
surrounding anatomy. In yet another embodiment, 
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multiplanar views such as orthogonal (sagittal, 
coronal, and axial) or oblique views, may be presented 
simultaneously with a surface-rendered organ and 
surrounding volume- rendered anatomy in either separate 
three-dimensional display windows or in a combined 
single three-dimensional display or window. 

A computer-controlled system 2 0 in accordance 
with the present invention is shown schematically in 
Fig. 4. The system 20 is generally as described in 
U.S. Patent No. 5,782,762, which is incorporated 
herein by reference. Briefly, a computer console 24 
is used to operate a scanner 22 to produce a series of 
two-dimensional, cross -sectional images of a selected 
three-dimensional object. The two-dimensional images 
are transferred from the console 24 to a computer 
graphics workstation 26 over a computer network 25. 
The graphics computer 2 6 is used to generate a three- 
dimensional rendering, such as a three dimensional 
surface rendering, of the selected object and to 
automatically identify potentially abnormal regions of 
the structure. The three-dimensional rendering is 
displayed on a computer monitor 28. Additionally, the 
displayed rendering can be recorded on a video 
recorder 3 0 or photographed and/or digitally recorded 
for future viewing. Various inputs, such as a 
computer mouse 27, are provided to the graphics 
computer to permit a user to manipulate the displayed 
imagery. 

The scanner 22 can be a General Electric HiSpeed 
Advantage Helical CT Scanner (GE Medical Systems, 
Milwaukee, WI) . For scanning a patient's lungs, the 
scanner is typically operated with an X-ray beam 
collimation of 3 mm, a pitch of 1:1, a 1 mm 
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reconstruction interval, a display f ield-of -view of 
25.6 cm, and a 512x512 matrix. For scanning a 
patient's colon, the scanner is typically operated 
with a beam collimation of 5 mm, a pitch of 2:1, a 
1 mm reconstruction interval, a display f ield-of -view 
of 40 cm, and a 512x512 matrix. These protocols 
typically result in approximately 200 thoracic CT 
images or 500 abdominal CT images, which occupy 
approximately 100 MB and 250 MB of disk storage space, 
respectively. However, the protocols can be modified 
to optimally acquire image data of specific organ 
systems, or to adapt to advances in scanner 
technology. Alternatively, MR scanning can be 
employed, or any future technology that generates 
image data. 

The image data can be stored in any of a variety 
of image formats. For example, the image data can be 
stored using the digital imaging and communications in 
medicine (DICOM) standard, or as raw binary slices, or 
in a variety of volume formats. For example, the 
image data can be stored in the computer memory in an 
internal data format which allows the image files to 
be saved as a single data volume instead of individual 
image files. The internal data format can also permit 
standard compression and uncompression techniques to 
be incorporated, thereby reducing computer disk 
storage requirements . 

An alternate arrangement of the system in 
accordance with the present invention is shown 
schematically in Fig. 11. The system of Fig. 11 is 
particularly well suited for use with a centralized 
high-speed graphics computer. The scanner 422, 
computer console 424, graphics computer 426, memory 
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412, mouse 427, monitor 428, and video recorder 430 
are essentially the same as the corresponding 
components of system 20. However, in system 42 0, the 
mouse 427, monitor 428, and video recorder 430 are 
connected to computer console 424. Additionally , ^ the 
graphics computer 42 6 is connected to the computer 
console 424 via a computer network 42 9 such as an 
outside network permitting remote connection of the 
graphics computer. The computer network 42 9 enables 
the graphics computer 426 to both receive image data 
from the computer console 424 and transmit the results 
of subsequent image data analysis to the computer 
console 424. Accordingly, the graphics computer 426 
can be located remotely from computer console 424. 
Further, graphics computer 42 6 can be used to analyze 
image data received from more than one computer 
console . 

A method for generating interactive, three- 
dimensional renderings of a selected structure having 
a lumen and indicating potentially abnormal regions of 
the structure in accordance with the present invention 
is generally set forth in Fig. 1. The method can be 
implemented by computer. The steps of patient 
preparation 32, acquisition of two-dimensional images 
33, formation of a three-dimensional volume 34, 
segmentation of a region of interest 35, creation of 
an isosurface of the region of interest 37a, and 
generation of a wireframe model of the isosurface 38a 
may be effected in the general manner described in 
U.S. Patent No. 5,782,762, which is incorporated 
herein by reference or any comparable methodologies 
and algorithms. Alternatively, instead of generating 
a wireframe model from an isosurface of a segmented 
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region of interest pursuant to steps 35, 37a and 38a, 
the wireframe model, can be generated at step 3 8b after 
a surface model is identified at step 37b from the 
three-dimensional volume formed at step 34. 

Patient preparation 32 will be dictated by the 
organ system to be examined. For example, if the 
patient's colon is to be examined, the patient is 
prepared using bowel cleansing and insufflation with 
gas to distend the colon. In one embodiment, a hand- 
operated air bulb may be used to distend the colon. 
The colon is inflated via a rectal catheter by 
squeezing the air bulb to instill puffs of room air 
into the colon. 

Alternatively, the colon may be inflated with 
carbon dioxide (C0 2 ) or other gas, such as nitric oxide 
(NO) , which exhibits a high lipid solubility and a 
high partial pressure gradient across the colonic 
mucosa. The combination of properties provided by 
carbon dioxide results in less uncomfortable abdominal 
symptoms for the patient following an examination. 
However, the high lipid solubility and high partial 
pressure gradient across the mucosa mean that the 
carbon dioxide is quickly absorbed by the patient. 
Therefore, the pressure of carbon dioxide within the 
colon must be continually monitored and maintained at 
a beneficial level during the course of an 
examination. Toward that end, an electronic carbon 
dioxide insufflator, such as those currently used for 
laparoscopic surgery procedures, is preferably used to 
instill the carbon dioxide at a constant flow rate 
(e.g., 1-15 liters/min) until a preset pressure (e.g., 
0.04 atm) is obtained. The insufflator then maintains 
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a constant pressure during the course of the 
examination. 

The patient may be placed on a low-residue diet 
for a 24 hour period preceding the collection of image 
data. In addition the patient can be given an oral 
contrast agent (barium-containing or iodine-containing 
mixture) to become incorporated in the fecal content 
to readily distinguish the stool from surrounding soft 
tissue structures, including the colon wall. The 
contrast agent may be used in conjunction with a 
cathartic agent to rid the bowel of the bulk of stool . 
The opacified stool may then be digitally removed from 
the CT image data in order to render an unimpeded 
colon. Without adequate bowel cleansing or the use of 
an oral contrast agent, stool and normal colon wall or 
colon lesions may be indistinguishable in computed 
tomography or other imaging modalities. For example, 
the contrast agent may comprise a barium-containing 
mixture or an iodine -containing compound for 
opacifying solid or liquid stool. In the case of 
rendering a patient's airways, the patient can be 
administered a bolus of non- ionic intravenous 
iodinated contrast agent to aid in distinguishing 
blood vessels from surrounding airways. 

Once the patient has been prepared, two 
dimensional, cross- sectional images of the selected 
structure are acquired at step 33 with the use of a 
scanner 422, such as a helical computed tomography 
(CT) scanner or magnetic resonance imaging (MRI) 
scanner. The two-dimensional images are arranged in 
computer memory 421 (i.e., random access memory (RAM) ) 
to create a three-dimensional data volume at step 34. 
To create isocubic volume elements (i.e., voxels), an 
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interpolation method, including but not limited to 
trilinear interpolation, may be applied. 

At step 35 , a region of interest is segmented 
from the three-dimensional data volume. The purpose 
of segmentation is to isolate a region of interest 
within the three-dimensional data volume prior to 
three-dimensional rendering. In general, medical 
image segmentation is complicated by image noise, 
partial volume effects, and the fact that similar 
intensity values are shared by different anatomical 
structures. However, when a thin-walled soft tissue 
structure encompasses a gas-filled lumen, segmentation 
of that structure can be effectuated by selecting the 
gas column as the region of interest. The gas column 
can be effectively segmented because the boundary 
between gas and soft tissue is relatively distinct and 
sharp. Further, the outside surface of the gas column 
corresponds to the inside surface of the organ of 
interest. Similarly, when a thin-walled soft tissue 
structure encompasses a contrast enhanced blood-filled 
lumen, segmentation of that structure can be 
effectuated by selecting the contrast -enhanced blood 
column as the region of interest. Accordingly, a 
simple thresholding or region growing technique, along 
with a morphological dilation away from air or blood 
vessel and towards soft tissue, can be utilized for 
segmentation purposes. 

However, in order to refine and to improve the 
accuracy of the segmentation process, especially as 
selected structures have varying characteristics 
(e.g., changing diameter, wall thickness, and/or X-ray 
attenuation values) , more sophisticated segmentation 
procedures can be employed. One such procedure 
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adaptively adjusts the threshold value defining the 
region of interest to more accurately segment specific 
sections of the organ during a region growing process 
as described in U.S. Patent No. 5,920,319 incorporated 
herein by reference. Alternatively, the accuracy of 
segmentation can be improved by refining (or 
deforming) the wireframe model generated in steps 3 8a 
and 38b of Fig. 1 using a geometric deformable model 
(GDM) at step 40b. 

Referring again to Fig. 1, when a patient's colon 
is to be rendered and a contrast agent has been used 
to opacify stool and/or residual water, the 
segmentation step 3 5 includes the step of digitally 
removing the opacified stool/water from the CT image 
data in order to render an unimpeded colon. Digital 
removal of opacified feces is difficult due to the 
fact that simple digital subtraction leaves behind 
edge gradients. The appearance of edge gradients is 
illustrated in Figs. 15-17. Fig. 15 is a schematic 
representation of a section of image data representing 
hypothetical CT image data showing the measured X-ray 
attenuation values associated with a two dimensional 
array of voxels. Soft tissue is represented by a 
value of 100, opacified stool by a value of 2 00, and 
air by a value of 0. In this simplified model, actual 
interfaces between different species is presumed to 
lie on the boundary between adjacent voxels. 
Accordingly, an interface 600 between soft tissue and 
opacified stool appears as the boundary between the 
voxels having a value of 10 0 and those having a value 
of 200. Similarly, an interface 601 between opacified 
stool and air appears as the boundary between the 



WO 00/32106 PCT/US99/28030 

voxels having a value of 200 and those having a value 
of 0, 

In reality however, sections of actual interfaces 
are likely to fall within certain voxels. The X-ray 
attenuation values measured for those voxels will be 
an average of the X-ray attenuations for the different 
species comprising the interface. Such volume 
averaging effects blur the interfaces. As illustrated 
in Fig. 16, the actual soft tissue/opacif ied stool 
boundary falls somewhere within the voxels having a 
measured value between 100 and 200. Likewise, the 
actual opacified stool/air interface lies somewhere 
within the voxels having a measured value between 200 
and 0. 

The effect of a simple digital removal scheme, 
whereby all values that are 150 or greater are set to 
zero, on the section illustrated in Fig. 16 is shown 
in Fig. 17. It is readily apparent that a ridge of 
voxels having a measured value of 90 is left behind 
and, therefore, will incorrectly be considered part of 
the soft tissue. In addition, although the true soft 
tissue/opacif ied stool interface lies somewhere within 
the voxels having a value of 120, the removal scheme 
automatically assumes that all of those voxels belong 
to the soft tissue. It will of course be appreciated 
that the edge effects encountered in real image data 
will be even more complex, due at least in part to 
inherent fluctuations in the measured values. 

In view of the foregoing, when opacified stool is 
to be subtracted from the image data, an adaptive 
thresholding technique (as previously described in 
U.S. Patent No. 5,920,319) and/or a geometrically 
deformable model approach can be used to model both 



WO 00/321 06 PCT/US99/28030 

the gas column and the adjacent stool. These 3D models 
can then be analyzed separately, or merged to create a 
contiguous colon model. For example, the geometric 
deformable models initialized separately in the gas 
column and adjacent stool can be allowed to evolve 
until the boundaries of the two models merge. 

Returning to Fig. 1, once the region of interest 
has been segmented at step 35, an isosurface of the 
region of interest is created at step 37a. The 
isosurface can be generated using a variant of a 
marching cubes algorithm. The isosurface is used to 
generate a wireframe model at step 3 8a. The wireframe 
model comprises a polygonal mesh that corresponds to 
the surface of the region of interest. Specifically, 
a plurality of vertices , each vertex having a 
coordinate position, are interconnected to produce the 
wireframe model. 

Alternatively, a waltzing cubes algorithm can be 
employed as shown in Fig. 12 to first identify a 
surface model at step 37b and then to generate a 
wireframe of the surface model at step 3 8b without the 
help of initial segmentation 35. The generated 
surface can be an isosurface, such as that produced by 
a marching cubes variant, or it can be a variable- 
thresholded surface. 

First, a voxel near the surface of interest 
within the volume of data is selected at step 500. 
This selection can be automated or explicitly 
identified by the user. Using this selected voxel, the 
first seed cell and its coordinate are identified at 
step 502. A cell is defined as the junction of eight 
adjacent image voxels, four in an upper image plane 
and four in a lower image plane, such that the eight 
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voxels form a 2x2x2 cubic cell. A cell coordinate 
defines its exact location within the volume of data. 
The first seed cell is defined as any cell near the 
selected voxel that produces a surface patch belonging 
to the surface of interest. The first seed cell can 
be determined by iteratively examining all cells 
within an increasing radius centered on the selected 
voxel. Visually, each successive iteration examines 
all cells that lie on the surface of a sphere centered 
on the selected voxel as the radius is increased, 
until the nearest surface patch producing cell is 
identified. This method is accurate but 
computationally slow. Alternatively, cells that lie 
along each direction of each coordinate axis (i.e. -X, 
X, -Y, Y, -Z, Z) centered on the selected voxel can be 
examined until the nearest seed cell is identified. 
This method is fast but requires a more exactly 
selected voxel. 

After the first seed ceil and its coordinate are 
identified, the coordinate is inserted into the 
waltzing cubes seed queue at step 504. A seed queue 
stores coordinates of seed cells that are added and 
removed in subsequent steps. 

At step 506, the seed queue is queried to see if 
it is empty. If the seed queue is empty, the waltzing 
cubes algorithm is terminated at step 520, since all 
cells that may produce a connected surface patch of 
the surface of interest have been analyzed and 
processed. If it is not empty, then the algorithm 
proceeds to the next step, and the main loop of the 
algorithm is entered, because all of the cells which 
may produce a connected surface patch have not been 
analyzed. Note that at the start of the algorithm, 
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the only seed coordinate in the seed queue is the 
first seed cell coordinate added at step 504. 

At step 508, a seed coordinate is removed from 
the seed queue. At step 510, using the seed cell 
coordinate, the waltzing cubes cell tree is queried 
for the corresponding seed cell. The cell tree 
identifies, organizes and stores all cells within the 
volume of data that belong to the surface of interest . 
Additionally, the cell tree maintains cell-to-cell 
vertex coherency data to ensure that exactly one 
calculation is made for each shared vertex that lies 
along the edge between surface patches in two adjacent 
cells . 

The cell tree is structured to contain a set of 
buckets, A bucket is defined as a linear array of 
cells where subsequently added cells are appended at 
the end of the array. A bucket search for a 
particular cell is done in reverse order, starting 
with the cell at the end of the array and working back 
to the beginning. Alternative search algorithms, such 
as binary search, can be used. However, the reverse 
order linear search is optimal. Each bucket in the 
cell tree represents all examined cells within a 
subset of the volume, called a cell block. The volume 
is divided into cell blocks of optimal size, the size 
depending upon the ratio of surface patch producing 
cells to the total number of cells within the block. 
An increase in the volume size results in an increase 
in the number of blocks because the block size remains 
the same. Typically, a cell block size of 16x16x16 
(4096 cells) is near optimal. 

At step 512, if the seed cell exists in the cell 
tree, it has been previously examined, so it is 
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ignored, and the loop continues by returning to step 
506. If the seed cell does not exist in the cell 
tree, it has not yet been examined and processed, so 
it is inserted into the cell tree at step 514. 

At step 516, a surface patch comprised of 
triangle primitives for the seed cell is produced. 
The surface patch can be generated by using the seed 
cell configuration and the corresponding triangulation 
table for that configuration. The triangulation can 
be an isosurface patch that has been pre-constructed 
for each cell configuration and stored in a 
triangulation table. Alternatively, the triangulation 
can be a variable- threshold surface patch that is 
created dynamically using local thresholds, adaptive 
thresholding or other criteria. The triangulation 
process uses the cell-to-cell vertex coherency data to 
reuse vertices computed by other cells and to create 
new vertices that are not yet computed. The 
triangulation may produce surface patches that are 
contained entirely within the seed cell or that span 
the seed cell and its adjacent cells in the cell tree. 

At step 518, the adjacent cell coordinates are 
determined for the seed cell using the seed cell 
configuration and the corresponding adjacency table 
for that configuration. An adjacent cell coordinate 
is the cell coordinate for the corresponding cell that 
lies adjacent to the seed cell. Each seed cell has up 
to 6 adjacent cells, and therefore, up to 6 adjacent 
cell coordinates. An adjacency table contains, for 
each seed cell configuration, a pre-analyzed list of 
adjacent cell coordinates that identify cells that 
each produce a surface patch that is connected to the 
surface patch produced by the seed cell. Each of 
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these adjacent cell coordinates is used to query the 
cell tree to determine whether it has already been 
examined. If it has been examined, the adjacent cell 
coordinate is discarded. If it has not been examined, 
the adjacent cell coordinate is inserted into the seed 
queue. The algorithm loops back to step 506. When 
the algorithm terminates at step 52 0, a wireframe 
model of a complete surface is produced. 

The waltzing cubes method enables the run- time 
memory required to create the surface of interest to 
be significantly reduced. Specifically, the memory 
required is directly proportional to the number of 
cells that must be analyzed. Since the waltzing cubes 
method divides the volume of data into cell blocks, 
entire buckets of cells may be pruned (deleted) once 
it is known that the cell data is no longer needed. 
Accordingly, since only those cells which produce 
surface patches that are a part of the surface of 
interest are examined, the waltzing cubes method 
typically examines a small fraction of the total 
number of cells within the volume, resulting in an 
algorithm that is fast and efficient. 

After the wireframe model has been generated at 
step 38a or 38b of Fig. 1, connectivity matrices are 
determined at step 39. The connectivity matrices are 
data structures which provide information regarding 
the connectivity between the vertices and polygons 
which comprise the wireframe model. The connectivity 
matrices are determined by traversing the polygon and 
vertex lists associated with the wireframe model, 
generating sets of immediate neighbor vertices and 
triangles associated with each vertex and polygon in 
the lists. For example, the vertex to polygon 
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connectivity matrix contains, for each vertex in the 
wireframe model, a list of all polygons that contain 
that vertex. Likewise, a polygon to polygon matrix 
lists all adjacent polygons for each polygon in the 
model . 

Using these connectivity matrices, the polygon 
list of the wireframe model is then re-ordered. The 
polygons generated using the variant of marching cubes 
are usually added to, and as such ordered in, the 
polygon list as the three dimensional volume is 
traversed. Accordingly, all of the polygons between 
the first and second images are generated first, 
followed by the polygons between the second and third 
images, and so forth. This polygon ordering is not 
intuitive, nor ideal for analysis or manipulation of 
the geometry at the local connectivity level. 
Consequently, in order to group vertices into 
populations and to analyze the populations, it is 
advantageous to re-order the polygons into the 
sequence by which they would be encountered while 
"traveling" through the lumen of the wireframe model. 
In addition, the re-ordering facilitates rapid three- 
dimensional renderings, since hidden objects or 
objects outside of the f ield-of -view can be easily 
identified and removed, thereby reducing the number of 
polygons rendered and thus increasing the rendering 
speed. 

After connectivity matrices are computed, a 
normal vector is calculated for each vertex in the 
wireframe model at step 40a of Fig, 1. The direction 
of each normal vector is perpendicular to a plane that 
is tangent to the isosurface at each such vertex, 
typically pointing away from the object or away from 
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the lumen of a body organ. The normal vectors at the 
respective vertices can be computed as the average of 
the normal vectors associated with each polygon 
connected to that vertex. The normal vector 
associated with a polygon is calculated using a vector 
cross product . 

The wireframe model is refined using a geometric 
deformable model at step 4 0b. In a geometric 
deformable model , a surface is treated as a wave front 
that propagates in space over time until a targeted 
boundary is reached. With respect to the wireframe 
model, each vertex of the wireframe model is moved 
along its normal vector direction at a speed that is a 
function of both the surface curvature and the doubt 
(or confidence) level of a boundary estimate at that 
point. The surface evolves under this type of motion 
until the targeted boundary is reached, at which time 
the speed of motion approaches zero. Formally, if 
S(0) is an initial smooth surface and S(t) is a. one- 
parameter family of surfaces generated by wave front 
propagation, the surface S should satisfy the 
following flow equation: 

M=(4>ff-V<j> • M)N (l) 

where H denotes the mean curvature, N is a unit 

normal vector, and $ is a positive dif f erentiable 
function defined in three-dimensional space that acts 
as a stopping function (i.e., <t> approaches zero as the 
surface nears the targeted boundary) . 
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A direct solution for this equation is difficult 
because of the topology changes and singularities that 
are created during the course of surface evolution. 
Accordingly, special techniques are typically used to 
facilitate convergence of Eqn. (1) . For example, a 
level -set technique can be used, such as the one 
proposed by Osher and Sethian in "Fronts propagation 
with curvature dependent speed: algorithms based on 
Hamilton- Jacobbi formulations", J. Computational 
Physics, 1988, 79; 12-49, which is hereby incorporated 
by reference. In the level-set approach, the surface 
S is embedded as a level surface ¥=0 in a three- 
dimensional scalar function The evolving family of 
surfaces S(t) is then generated from the level surface 
of an evolving family of scalar functions ¥{t) . As 
shown in Fig. 3, evolving contours are treated as 
zero- level curves of the evolving function F. By 
shifting the scalar function F downward from time tl 
to time t2 and extracting the zero level curves, the 
net effect of shrinking the circular contour (Fig. 
3(a)) is achieved (Fig. 3(b)). If F is shifted down 
even further, the circular contour will appear to 
split into two smaller segments. 

For the embedded level -surf ace to evolve 
according to Eqn (1) , the scalar function W should 
evolve according to the equation: 




(2) 
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The divergence operator di\ 



IIVTilJ 



computes the mean 



curvature H of the embedded level surface from the 
scalar function ^. The constant, v, determines if the 
surface evolves inwardly or outwardly. The stopping 
function, <p, can be defined as: 



l+liVG 0 */]| 



where G 0 is a Gaussian filter whose characteristic 
width is o and * denotes a convolution operator. 
Images convoluted with the Gaussian filter before 
gradient computation are smoother and produce more 
accurate results. The purpose of the Gaussian filter 
is to smoothly force the evolution speed to zero at 
points where high intensity gradients exist. 

More generally, the stopping function can be 
characterized as a doubt function that approaches 0 
when there is no doubt that the wave has reached a 
boundary and 1 otherwise. Using this characterization 
better stopping functions can be designed so that they 
incorporate not only edge gradient information, as in 
Eqn(2), but also other critical information such as 
noise model and scale level. Furthermore, Bayesian 
statistical models can ve used to properly define the 
doubt function since many factors contribute to one's 
doubt (or confidence) about any point being on a 
boundary . 

Traditionally, a major problem associated with 
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the use of geometric deformable models has been poor 
computational efficiency due to the fact that multiple 
iterations are required at numerous grid points. One 
approach to improve the computational efficiency has 
been the use of narrow-band or tube methods as 
described by Malladi et al . in "Shape modeling with 
front propagation: a level set approach", IEEE Trans. 
Pattern Analysis and Machine Intelligence, 1995/ 
17:158-175, which is hereby incorporated by reference. 
Although useful in some applications, the narrow-band 
method is inefficient when dealing with large 
wireframe models, such as those representing a colon. 
Accordingly, special constraints may need to be added 
to reduce the processing time for large objects. In 
one embodiment, a three-dimensional region -growing 
model is used as a starting point for the deformable 
model approach. Alternatively, the deformable model 
approach may be applied to a low resolution dataset to 
produce an intermediate wireframe model . The 
intermediate wireframe model is then translated to a 
higher resolution dataset. Further, since the 
original volume of data typically has a high spatial 
resolution, the deformable model approach can be 
applied to a subsampled volume (e.g., one-eighth of 
the original volume of data) to produce an 
intermediate wireframe model without significantly 
affecting the shape of the rendered image. The 
intermediate wireframe model is then applied as a 
starting point for processing of the complete volume 
of data. 

The vertices of the wireframe model can be 
analyzed and grouped into populations having abnormal 
wall structure as shown at steps 41-45 in Fig. 1. 
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The wall thickness of the structure at each 
vertex is measured at step 41- The wall thickness at 
a given vertex can be determined by measuring the 
distance from the inner surface of the selected object 
to its outer surface or some other position between 
the inner and outer surfaces. For body parts such as 
the colon or airways, the selected structure's wall 
is composed of soft tissue and the lumen of the 
structure comprises gas. Accordingly, the inner 
surface corresponds to a gas/soft tissue interface. 
Further, the colon and airways are typically 
surrounded by fat, gas, or other non-soft tissue 
material (i.e., contrast -enhanced blood vessels, bone, 
or calcification) . Accordingly, the outer surface 
corresponds to the soft tissue/gas, soft tissue/bone, 
soft tissue/fat, or other such interface. The X-ray 
attenuation factors for room air (approximately 
-1024 HU to -425 HU) , fat (about -200 HU to -20 HU) , 
and other non-soft tissue structures such as contrast- 
enhanced blood vessels and bone (approximately >100 
HU) are distinct from the soft tissue component (about 
20 HU to 100 HU) of the airways or colon wall. 
Accordingly, the wall thickness at a vertex on the 
wireframe model can be calculated from the volume data 
of the selected organ, which can be measured at 
regularly spaced intervals along each normal vector. 
Alternatively, wall thickness can be estimated by 
analyzing differential characteristics of the 
intensity profile along the normal direction. For 
example, the transition from gas-filled lumen to wall 
tissue and then from wall tissue to fat or other 
software tissue can be captured by finding the first 
two peaks in the gradient of the intensity profile. 
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Referring to Fig. 6, a wall thickness at a vertex 
can be calculated by analyzing the intensity profile 
along the normal vector originated at the vertex. An 
intensity profile of a vector is defined as an array 
of interpolated x-ray attenuation factors (voxel 
values) along regularly spaced intervals along a 
vector originating at the vertex. The analysis of the 
intensity profile may include several heuristics 
depending upon the intensity profile type. One simple 
heuristic is to measure the thickness as the distance 
between the vertex and the point at which the 
attenuation factor drops below that of air (-425 HU) 
as seen in Fig. 6a. Another heuristic is to measure 
the thickness as the distance between the vertex and 
the point at which the attenuation factor exceeds 
above that of soft tissue (60 HU) as seen in Fig. 6b. 
Yet another heuristic is to measure the thickness as 
the distance between the vertex and the point at which 
there is a local minimum as seen in Fig. 6c. These 
heuristics are often too basic and are insufficient to 
accurately determine complex intensity profiles such 
as that in Fig. 6d. Therefore, a more mathematical 
analysis of the intensity profile is required for 
accurate wall thickness calculation. The advanced 
analysis include differential characteristics of the 
intensity profile, discrete first /second/third degree 
gradiant profiles, polynomial curve fitting along the 
intensity profile and solutions to differential 
equations based on the curve, and other knowledge 
based methodology. For optimal wall thickness 
measurement, a weighted combination of basic and 
advanced analysis is employed, and the results are 
averaged and normalized to reduce inaccuracies. 
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Alternatively, the wall thickness at the vertex on the 
wireframe model can be calculated from the volume of 
data of the selected organ near the vertex. For 
example, another approach to estimate wall thickness 
is to classify the voxels in the vicinity of the 
segmented lumen surface into a small number of 
intensity groups. The shape of the resulting groups 
can be analyzed to determine the voxels that belong to 
wall and the wall thickness can then be estimated from 
these voxels . 

Since the wall thickness of body structures such 
as the colon may be inadvertently or artificially 
increased, such as during musculature contraction of 
the colon at certain positions along the colon 
structure, many of the regions of abnormal thickness 
may not be indicative of true lesions. Instead, the 
thickened regions may represent normal structural 
elements, such as regions where the colon is 
contracted or where it touches solid organs. 
Accordingly, it can be beneficial to further analyze 
and characterize each vertex associated with abnormal 
thickness to refine the list of potentially abnormal 
vertices. Additional local shape features, computed 
at step 42, can be used to characterize and associate 
vertices as being abnormal . 

For example, a local convexity value at each of 
the surface vertices can be determined at step 42. 
The local convexity values can be determined for all 
vertices, or for only those vertices associated with 
abnormal wall thickness. For each vertex, its 
neighboring vertices within a connected neighborhood 
of a pre -determined size (e.g., 3), is determined 
using the connectivity matrices. A local convexity 
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value for each vertex, v, is computed as: 



wherein D(vJ is the distance from the 1 th neighboring 
vertex to a plane through v and perpendicular to v's 
normal vector, and n is the number of neighboring 
vertices. Vertices associated with a convexity value 
of greater than a pre-defined maximum {e.g., >2) can 
be considered abnormal. 

In addition, the local curvature at each vertex 
in a population can also be determined at step 42 . 
The curvature reflects another shape feature of the 
object's surface at a given vertex. Vertices 
associated with broad curvatures indicate regions that 
are relatively flat. Accordingly, if the user wishes 
to identify only regions having thicknesses that vary 
significantly over short distances, those vertices 
with small curvatures may be labeled as abnormal. 

The local curvature at each vertex in a 
population can be determined by summing the scalar 
distance between each of a user-determined number of 
levels, N, of adjacent vertices and the plane 
perpendicular to the normal of the vertex. The result 
is then normalized by the surface area outlined by the 
N th level of adjacent vertices. Because the direction 
of the curvature is ignored, the technique is able to 
detect areas of small curvature. 
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Alternatively, local curvature can be computed at 
each vertex from the wireframe model represented as a 
level surface. In particular, as shown in Fig. 13, 
the mean curvature at each vertex of the wireframe 
model is determined at step 550. The mean curvature H 
(i.e., the average of the two principal curvatures) of 
the level surface W is given by: 



7 ? ? J/ 



The Gaussian curvature K is similarly determined 
according to the equation at step 552 : 



Each vertex of the wireframe model can be classified 
as one of four types at step 554. The point is 
classified as elliptic if K(p)>0, as hyperbolic if 
K(p)<0, as parabolic if K{p)=0 and H(p)*0, or as 
planar if K(p)=0 and H(p)=0. This classification 
scheme enables the vertices to be grouped into patches 
having vertices having the same characteristic shape 
at step 556. Since polyps protrude into the colon 
lumen, only elliptic patches are labeled as 
representing likely polyps at step 558. 

Additionally, elliptic patches that point 
outwardly from the colon are eliminated at step 560 
using the signs of the two principle curvatures, k x and 
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k 2 . The principal curvatures, k x and k 2 , are derived 
from the mean and Gaussian curvatures to be 



are both negative (or positive depending on coordinate 
system used) does the elliptic patch represent a 
potential lesion pointing inward with respect to the 
lumen . 

At step 44, vertices representing an abnormality 
are selected. Those vertices meeting selected 
criteria can be identified as abnormal vertices. For 
example, vertices having abnormal thickness and/or 
convexity and/or curvature criteria may be identified 
as abnormal vertices. In a specific application, the 
wall thickness values, local convexity values, and 
local curvature values can be used, independently or 
in conjunction with each other, to .either accept or 
reject vertices as being abnormal. The individual 
parameters can be combined using a logical AND 
operation. 

At step 45, the vertices on the wireframe model 
associated with abnormal structure (i.e., having a 
combination of associated abnormal wall thickness 
and/or abnormal shape, such as abnormal local 
convexity and/or abnormal local curvature) are grouped 
into populations. The re-ordered connectivity 
matrices are used to determine if vertices associated 
with abnormal parameters are directly connected to 
other vertices that are also associated with abnormal 
parameters. Accordingly, each formed population 
represents a potential abnormal lesion. In addition, 
each population can be further analyzed and 
characterized by its size, dimensions, or other 
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statistical quantities . 

A cross sectional area (CSA) of surface model is 
defined as the area outlined by the contour formed by 
the interaction of the surface model 900 with an 
arbitrary CSA plane 902 as shown in Fig. 8. The cross 
sectional area at the site of a lesion may be used to 
calculate the degree of stenosis, or narrowing, caused 
by the lesion at that location. The cross sectional 
area may also be used with a central path algorithm to 
determine areas of stenosis in a lumen of an organ. 
The surface model is represented as a wireframe mesh 
of triangles. Using the connectivity matrices, all 
triangles in the wireframe which intersect the CSA 
plane are identified as shown in Fig. 9a. For each 
triangle, edges of the triangle which intersects the 
CSA plane are identified, and intersecting vertex 
point between each edge and the CSA plane are 
computed. The set of these intersecting vertex points 
represents an approximation of the cross sectional 
area which is to be computed, and it is defined as the 
CSA contour. The CSA contour lies completely in the 
CSA plane as shown in Fig. 9b. The CSA contour 
vertices are used to triangulate a set of adjacent 
triangles that span the entire contour. The 
triangulation may simply define the edges of each 
triangle as consecutive contour vertices as shown in 
Fig. 9c. Alternatively, a centroid point for the 
contour may be computed and used for a more accurate 
triangulation as shown in Fig. 9d. Because the 
generated triangles span the contour, the cross 
sectional area of the contour is simply defined as the 
sum of the areas of each triangle within the contour. 
This method is a numeric approximation that is 
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sufficiently close to the true cross sectional area. 

Each population is further analyzed according to 
aggregate shape features at step 4 6 to reduce the 
occurrence of false positive findings. Since the 
structure of body organs may appear abnormal, either 
inadvertently or artificially, such as during muscular 
contraction of the colon at certain positions along 
the colon's structure, many of the abnormal 
populations may not be indicative of true lesions. 
Instead, these populations may represent normal 
structural elements, such as regions where the colon 
is contracted or where it touches solid organs. 
Accordingly, it may be advantageous to further analyze 
and characterize each population identified as having 
an abnormal structure to further refine the list of 
potential lesions and to increase the likelihood that 
a detected population represents a true abnormality. 

For example, the shapes of the populations can be 
analyzed, characterized, and accepted or rejected as 
being abnormal as illustrated in detail in Fig. 2. At 
step 120 a centroid for each population is calculated. 
The centroid can be computed as a weighted sum of the 
positions of each vertex within the population. The 
individual vertices can be weighted by their 
associated wall thickness values or other parameters. 
Once the centroid has been determined for a 
population, the height and normal vector for that 
population can be determined. 

A population normal vector is calculated for each 
population at step 121. The population normal vector 
can be computed as a sum of the normal vectors at each 
of the vertices within the population, and the 
individual vertex normal vectors can be weighted by 
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their corresponding wall thickness values or other 
parameters . 

The convexity value, (C> , for each population is 
determined at step 122. The convexity is a measure of 
the direction and magnitude of the shape of the 
surface of the population. As shown in Fig. 5, the 
convexity of a population is computed as the sum of 
the distances 94 from vertices 95 in the population to 
a plane 96, perpendicular to the population normal 
vector 97 passing through the centroid 98, according 
to the equation: 

C=I>(v.) 

1 = 1 



wherein D(v f ) is the distance from the 1 th vertex of 
the population to the plane 96 and n is the number of 
vertices 95 within a predetermined space relative to 
the centroid. Since the normals at each vertex are 
typically assigned to point away from the center of 
the object, or the lumen of the object, a population 
which protrudes into the lumen of the object would 
have a positive convexity value, whereas a population 
that projects away from the lumen would exhibit a 
negative convexity value- When the structure is a 
colon, populations with a negative convexity value are 
excluded from being considered abnormal, since 
abnormal colon masses tend to project into the lumen 
of the colon and would have a positive convexity 
value . 

Further, the higher the magnitude of the 
convexity value, the greater the slope of the surface 
of the population. When the structure is a colon, 
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only populations having a positive convexity value 
above a minimum value are reasonably expected to be 
potential lesions, since some cancerous colon masses 
are generally manifested by steeply sloped growths 
that protrude into the lumen of the colon. 

The height (H) of each population is determined 
at step 123. The height is calculated as the 
difference between the distance of the vertex farthest 
from a plane 96, perpendicular to the population 
normal 97 and passing through the centroid 98, and the 
distance of the vertex closest to that plane according 
to the equation: 



H = MAX D{vJ - MIN D(vJ . 



The height provides an indication as to whether a 
population is reasonably expected to be a true lesion 
since populations with large heights are more likely 
to represent abnormal masses . 

The principal curvatures, k x and k 2 , can also be 
used to reduce the likelihood of false positive 
identifications. Generally, polyps are nearly 
symmetrical and, accordingly, patches representing 
true polyps are likely to arise when the two principal 
curvatures on the top of a potential lesion are 
similar. Further, normal colon haustra have ridge- 
like shapes and, therefore, potential lesions having 
principal curvatures, k x and k 2 , which are 
substantially different are eliminated as likely 
polyps at step 562. 

Other shape features may be used to include or 
exclude populations in the list of potential lesions. 
For example, if a population has a long axis dimension 
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that is greater than a multiple (e.g., 10 times) of 
the short axis dimension, and the short axis dimension 
is, for example, less than 5 mm wide, then the 
population might be considered to be normal. Further, 
a potential lesions "roundness" could be quantified 
and used to rank potential lesions. Likewise, 
weighting wall thickness determinations with shape 
features may provide valuable heuristics for 
classifying potential lesions. Accordingly, the shape 
features discussed herein are included only by way of 
example, and are not intended to limit the scope of 
the invention. 

Referring again to Fig. 1, after the populations 
with abnormal wall structure (i.e., wall thickness, 
convexity, or other shape features) have been 
identified and characterized, populations that fail to 
meet certain criteria are eliminated as potential 
lesions at step 48. 

At step 50, those populations which are still 
considered to represent potential lesions are sorted 
and arranged in an electronic listing. The listing 
can be sorted according to such properties as size, 
convexity, curvature, height, mean wall thickness, 
standard deviation of wall thickness, other 
statistical quantities, or a combination of such 
parameters (e.g., the product of the group convexity 
value, C, and the group height value, H) . Each 
population in the listing could be linked to a three- 
dimensional camera position that best illustrates it 
in the three-dimensional display. Accordingly, the 
user can choose a population from the listing and 
thereby be presented with a view of the abnormal area 
as it appears in the three-dimensional rendering. One 
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means of linking the population to a three-dimensional 
rendering camera position is by providing the user 
with a three-dimensional camera view of the population 
that is parallel to the population normal and centered 
on the centroid of the population, and set at a chosen 
distance from the centroid. In one embodiment, the 
user is presented with two views of the selected 
population, one from an external perspective and the 
other from an internal perspective (i.e., from within 
the lumen of the structure) . Lesion locations, either 
identified manually by a user or by the above 
algorithms can also be used to generate lesion maps, 
i.e., two dimensional or three dimensional renderings 
of entire colon with lesion sites marked. 

A three-dimensional rendering of the wireframe 
model is displayed at step 51. In one embodiment, the 
user can navigate through a three-dimensional 
rendering of the wireframe model which is displayed on 
the monitor 28, The detected abnormal populations are 
displayed with a contrasting color scheme to 
distinguish them from the remainder of the rendered 
structure, so that the abnormal populations can be 
quickly and easily identified by human visual 
inspection. Further, the color scheme can reflect 
variations in wall thickness, convexity, or curvature, 
or some other parameter. For example, the normal 
portions of the rendered structure can be displayed in 
shades of pink while the potentially abnormal 
populations are displayed in a rainbow of colors. 
More specifically, vertices could be assigned colors 
ranging from dark blue (for vertices with minimum 
abnormal thickness) to bright blue (for vertices with 
maximum abnormal wall thickness) to provide a stark 
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contrast to the pinkish color assigned to the normal 
portions of the rendered structure. 

In an alternate embodiment, the user is presented 
with a volume rendered display centered on the 
centroid of a selected abnormal population. For 
example, a volume -rendered view of a subvolume of user 
defined size centered on the centroid of the 
population can be displayed for visual inspection. 
Volume rendering is a visualization technique that 
creates a three-dimensional image using all of the 
volume data, rather than only an extracted surface 
representation. Traditionally, volume rendering is 
accomplished by projecting a series of rays from the 
user's viewpoint through the data volume. The color 
at each point (i.e., pixel) in the resulting three- 
dimensional image is determined by calculating a 
weighted sum of all voxels along each ray. This 
weighting factor is determined by a transfer function 
that is computed during a process called 
classification. Additionally, voxels belonging to a 
lesion may be selectively enhanced with color and/or 
opacity values. 

Altering the transfer function during volume 
rendering can reveal different characteristics of the 
rendered data and, thus, it is often desirable to 
interactively reclassify and re-render the data. 
However, the rate at which images are rendered can be 
slow for data volumes as large as the ones typically 
used in medical imaging. Accordingly, it can be 
beneficial to use texture mapping to dramatically 
increase the rate at which volume rendering is 
performed. In texture mapping, volume data is 
considered to be a series of two-dimensional textures 
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that are interpolated, projected, and displayed 
rapidly on a series of polygons using specialized 
computer hardware . 

The texture mapped volume rendering method uses 
high-speed texture memory and blending hardware 
available on a Silicon Graphics' computer workstation 
to approximate the results of the more computationally 
expensive ray-casting algorithm. The volume data is 
first loaded into texture memory. The textures are 
then semi -transparently mapped onto a series of 
parallel two-dimensional rectangles that correspond to 
slices of the volume. The texture mapped rectangles 
are then drawn and blended to allow the user to see 
through the volume. The colors and transparencies of 
the resulting three-dimensional object are controlled 
by color and opacity transfer functions . The entire 
volume can then be viewed by stacking the texture 
mapped rectangles in the proper order and looking 
toward the stack. Since viewing the stack along one 
of its edges does not optimally display the volume, a 
new stack is generated which is orthogonal to the 
original stack. This new stack is then used to 
display the volume along an edge. Likewise, for any 
additional viewing angles, additional stacks may be 
required. Generating the stacks is a dynamic process 
and each stack can be generated on an as needed basis. 
Texture mapping can use standard two-dimensional 
texture memory or faster three-dimensional texture 
memory . 

in yet another alternate embodiment, upon 
selection of a population from the listing, the user 
can be presented with a three-dimensional, surface- 
rendered wireframe model of the population embedded 
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within a volume -rendered display using texture memory 
techniques. Such a display is useful to show the 
three-dimensional relationships between the wireframe 
model and the volume data. Embedding a surface- 
rendered model inside a volume rendering can be 
accomplished using texture memory since both 
techniques use semi-transparent or opaque surfaces to 
generate their imagery. However, if the surface 
rendering is semi-transparent, care must be taken to 
ensure that all objects are blended in the proper 
order (typically, from back to front) . All or part of 
the volume can be rendered. For example, to view a 
small structure using volume rendering, a subvolume of 
data surrounding that structure can be rendered, as 
opposed to displaying the entire volume of data. The 
point about which the subvolume rendering is performed 
can be selected manually by the user, or automatically 
by the above algorithms . It should also be 
appreciated that the multiplanar views and volume 
rendering can be combined in a similar manner. 

If the user wants to view the results of the 
various rendering techniques using separate display 
windows, the user can clone the display window 2 00 so 
that the two (or more) display windows share the same 
data. However, each display window could render using 
different techniques {i.e., multiplanar display, 
surface rendering, volume rendering) . Accordingly, 
the user could open a single window with all three 
rendering techniques and then clone the window three 
times to view each technique separately. Changes made 
to the data in any one of the connected windows would 
be propagated to all the cloned windows. Camera 
orientations could be synchronised between the 
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separate windows to lock the views together. Position 
tracking techniques could show the camera position of 
one viewer in another, so that the user could follow 
the path of a three-dimensional camera moving inside 
the structure in one display window from an external 
perspective using a cloned display window with another 
three-dimensional camera from an overhead position. 

An example of a computer console display 100 
useful for displaying a three-dimensional rendering of 
a structure is shown in Fig. 7, The display 100 
comprises a first window 101 for displaying a volume 
rendered image of the structure and a second 
window 102 for displaying a surface rendered image of 
the structure. Alternatively, the second window 102 
can display a surface rendered image of the structure 
embedded within a volume rendered display / as 
described above. In addition, three multiplanar 
windows, 103, 104, and 105, are provided in a third 
window for displaying axial, coronal, sagittal, or 
oblique two-dimensional slices through the structure 
at a user-defined point, or at the centroid of a 
chosen population. 

Alternatively a single window display 200, as 
shown in Fig. 10, can be used to display a three- 
dimensional rendering of the structure. The single 
window display comprises a single three-dimensional 
display window for presenting a combination of 
multiplanar images 203, 204 and 205 with a volume- 
rendered image 207 centered on a point located within 
a surf ace -rendered image 209. The display presents a 
holistic view of all the data that can aid in the 
visualization and understanding of the inter- 
relationships of the various rendering techniques 
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(i.e., surface rendering, volume rendering, and 
intersecting two-dimensional planes) . The various 
rendering techniques can exist together because they 
are all based on the same patient coordinate system 
supplied by the image scanner, and they are all 
rendered using shared three-dimensional graphics 
techniques (texture memory) and a common graphics 
library. The intersecting planes can be rendered as 
opaque textures (as opposed to semi-transparent 
textures used in volume rendering) mapped onto two- 
dimensional rectangles using texture mapping and drawn 
in their proper orientation. The user can 
interactively scroll through slices parallel to any of 
the intersecting planes and can view any combination 
of planes (e.g., axial and/or sagittal and/or coronal 
planes) , or the user can create and view an oblique 
slice plane. The planes can be rendered together with 
the surface data to show the exact relationship 
between the wireframe model and the volume (by viewing 
the intersections and orientations of the surface 
rendered wireframe model and the two-dimensional 
slices) . They can be viewed together because the 
planes are rendered as any other kind of surface 
(i.e., a flat, two-dimensional rectangle). 

The method and system of the present invention 
can also be used to guide needle biopsy procedures of 
diagnosed lesions as illustrated in Fig. 14. One 
example is biopsy guidance of cancerous lymph nodes in 
patients with lung cancer via transbronchial needle 
aspirations (TBNA) . At steps 800 and 802, the 
patient 1 s airway, lymph nodes, blood vessels, and any 
cancerous masses are segmented and rendered in a 
three-dimensional display. The rendering is performed 
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as described above. Of course/ the structures must be 
accurately rendered in order to provide a reasonable 
likelihood that the procedure will be successful. An 
alternative approach to automatically rendering 
segmenting the lymph nodes and/or the cancerous masses 
from the volume data, spherical "targets" can be * 
manually placed within lesions at lesion locations 
previously identified on the multiplanar data images 
to indicate the position and approximate size and 
shape of the targeted object. Alternatively, irregular 
targets can be modeled by having the user free-hand 
draw regions of interest on the multi -planar images 
from which three-dimensional targets can be rendered. 

Once the lymph nodes and cancerous masses have 
been identified and rendered, each lymph node and 
cancerous mass which is to be biopsied is labeled as a 
"targeted structure" at step 804. A zone, or striking 
distance, is then defined around each of the targeted 
structures in step 806. The zone represents a volume 
of space surrounding the targeted structure which is 
sufficiently close to the structure to enable a biopsy 
needle to reach the targeted structure and biopsy 
tissue from the targeted structure. At step 808 , 
blood vessels and other mediastinal tissues may be 
rendered. Rendering of the mediastinal tissues is 
performed because it is important to avoid puncturing 
major blood vessels during the needle biopsy 
procedure . 

An endoscopic biopsy procedure, such as TBNA, is 
performed with the actual biopsy needle retracted 
within the instrument channel of an endoscope while 
the endoscope is inserted in the airway. As soon as 
the endoscope tip is within striking distance of a 
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pre-selected target, the needle is extended into the 
target and then retracted back into the endoscope's 
instrument channel, hopefully containing a sample of 
tissue. Unfortunately, doctors often make "blind" 
biopsies into tissue since they cannot visually see 
through an opaque organ, hence the purpose of 
computer-simulated guidance. "Striking distance 7 ' is 
defined as the maximum distance from the target that 
the endoscope tip can located in order to obtain an 
adequate sample when the needle is extended. In 
clinical practice, this distance should be at most 
half the length of the needle- -% cm for a 1 cm needle- 
-so that when the needle is extended a sample of at 
least % the needle length is obtained. Also, both the 
scope's position and orientation must be taken into 
account to decide if the needle will be able to 
effectively biopsy the target, otherwise, if the scope 
tip is within striking distance but pointed away from 
the target, then the extended needle will not hit the 
target. If the tip is not pointed directly at the 
target but is pointed at an oblique angle to the 
target, the needle may glance off the side of the 
target and not obtain an adequate tissue sample. And 
if the target's diameter is smaller than the needle 
length, there is a chance that the tip may completely 
penetrate the target and biopsy tissue on the other 
side of the target. 

The simulated endoscope containing a biopsy 
needle within its instrument tract is then inserted 
into and moved within the patient's airway at step 
810. As the simulated endoscope moves through the 
airway, the position of the tip of the needle is 
monitored at step 812. The monitoring can be done 
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either on a continuous basis or intermittently. If 
desired, the movement of the simulated endoscope tip 
through the airway can be recorded and documented at 
step 811. 

Once the position of the simulated endoscope tip 
with its needle is determined, a rendered image of the 
needle tip within the rendered image is displayed at 
step 814. Accordingly, the physician performing the 
biopsy procedure can visually monitor the movement of 
the simulated needle through the airway in relation to 
surrounding anatomy. In order to more realistically 
simulate the view that would be obtained using a 
conventional bronchoscope, the displayed image can be 
altered to reflect the distorted camera view of 
conventional bronchoscopes. In addition to displaying 
a rendered image of the simulated needle tip within 
the airway, an external view of the airway showing the 
position and orientation of the needle can be 
simultaneously displayed. 

At step 816, it is determined whether the needle 
tip is within the zone, or within striking distance, 
of a targeted structure. If the needle tip is not 
within the striking distance of a targeted structure, 
then the distance and trajectory for the needle to 
reach a targeted structure is determined at step 818. 
The needle is then repositioned within the airway at 
step 810 using the calculated trajectory and distance 
as a guide for the movement of the needle . 

However, if the simulated needle tip is within 
the striking distance of a targeted structure at step 
816, then a ready-to-strike target proximity indicator 
is issued at step 820. The indicator serves to signal 
the physician to biopsy the targeted structure, at 
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step 822 . There are four types of target proximity 
indicators: (1) "getting close" indicator that 
increases in intensity as the endoscope tip approaches 
the target, (2) "strike zone" indicator that tells 

5 when the endoscope tip is within striking distance of 

the target and in the proper orientation to obtain an 
adequate tissue sample, (3) "too close" indicator that 
signals when the scope tip is too close to the target, 
and (4) "close, but no cigar" indicator that tells 

10 when tip is close enough to strike but is not pointing 

in the right direction for obtaining a good sample. 
In addition to these four indicators, a fifth type or 
alarm can be activated when the needle is in danger of 
penetrating vital organs near the target and putting 

15 the patient at risk. 

It will be recognized by those skilled in the art 
that changes or modifications may be made to the 
above -described embodiments without departing from the 
broad inventive concepts of the invention. It should 

20 therefore be understood that this invention is not 

limited to the particular embodiments described 
herein, but is intended to include all changes and 
modifications that are within the scope and spirit of 
the invention as set forth in the claims. 

25 



47 



WO 00/32106 PCT/US99/28030 
WHAT IS CLAIMED IS; 

1. A system for interactively displaying three 
dimensional structures comprising: 

a. volume formation means for forming a 
three-dimensional volume of data from a 
series of two-dimensional images 
representing at least one physical property 
associated with a three dimensional body; 

b. segmentation means for segmenting a region 
of interest from the volume of data based on 
selected values of the physical property 
representing the region of interest; 

c . means for producing a wireframe model of the 
segmented region of interest, the wireframe 
model comprising a plurality of vertices, 
each vertex having a coordinate position; 

d. means for refining the wireframe model by 
adjusting the coordinate positions of the 
vertices to more accurately represent the 
region of interest; and 

e. rendering means for rendering the wireframe 
model in an interactive three-dimensional 
display, therein producing a virtual three- 
dimensional environment . 



2. The system of Claim 1 wherein the segmentation 

means approximates the region of interest and the 
refinement means adjusts the coordinate positions 
of the vertices along a normal vector associated 
with each vertex. 



3 . The system of Claim 2 wherein the refinement 
means adjusts the coordinate positions of the 
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vertices along a normal vector associated with 
each vertex as a function of a surface curvature 
at the coordinate position. 

4 . The system of Claim 2 wherein the refinement 
means adjusts the coordinate positions of the 
vertices along a normal vector associated with 
each vertex by determining an inner wall position 
and an outer wall position from a first and a 
second degree derivative associated with the 
coordinate positions of the vertices. 



5. The system of Claim 1 wherein the segmentation 
means, the wireframe producing means, and the 
refining means are located remotely from the 
volume formation means. 



6. A system for interactively displaying three 
dimensional structures comprising: 

a. volume formation means for forming a 
three-dimensional volume of data from a 
series of two-dimensional images 
representing at least one physical property 
associated with a three dimensional body; 

b. segmentation means for segmenting a region 
of interest from the volume of data based on 
selected values of the physical property 
representing the region of interest; 

c. isosurface creation means for creating an 
isosurface of the segmented region of 
interest ; 

d. model means for producing a wireframe model 
of the segmented region of interest; and 
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e . rendering means for rendering the wireframe 
model in an interactive three-dimensional 
display, therein producing a virtual three- 
dimensional environment . 



7 . The system of Claim 6 wherein the isosurf ace 
creation means comprises: 

a. means for identifying a seed voxel from the 
segmented region of interest; and 

b. means for producing an isosurf ace patch 
associated with the seed voxel. 

8. The system of Claim 6 wherein the segmentation 
means, the isosurf ace creation means, and the 
wireframe producing means are located remotely 
from the volume formation means. 

9. A system for interactively displaying a three- 
dimensional rendering of a structure having a 
lumen and indicating regions having abnormal 
structure compr i s ing : 

a. volume formation means for forming a three- 
dimensional volume of data from a series of 
two-dimensional images representing at least 
one physical property associated with the 
three-dimensional structure ; 

b. segmentation means for segmenting a region 
of interest from the volume of data based on, 
a selected value of the physical property 
representing the region of interest; 

c. wireframe generation means for generating a 
wireframe model of the segmented region of 
interest ; 
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vertex grouping means for grouping the 
vertices of the wireframe model into regions 
having abnormal structure; 
analyzing means for analyzing the regions 
having abnormal structure; and 
rendering means for rendering the wireframe 
model in an interactive three-dimensional 
display to indicate the regions having 
abnormal structure . 

10. The system of Claim 9 wherein the analyzing means 
comprises means for evaluating two principal 
curvatures associated with each point in an 
elliptical region. 

11. The system of Claim 10 wherein the analyzing 
means comprises a means for identifying ridges. 

12 . The system of Claim 10 wherein the segmentation 
means, the wireframe generating means, the vertex 
grouping means, and the analyzing means are 
located remotely from the volume formation means . 

13. A computer- implemented method for interactively 
displaying a three-dimensional rendering of a 
structure having a lumen and indicating regions 
having abnormal structure comprising the steps 
of: 

a. forming a three-dimensional volume of data 
from a series of two-dimensional images 
representing at least one physical property 
associated with the three-dimensional 
structure ; 
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b. segmenting a region of interest from the 
volume of data based on a selected value of 
the physical property representing the 
region of interest ; 

c. generating a wireframe model of the 
segmented region of interest, the wireframe 
model comprising a plurality of vertices, 
each vertex having a coordinate position; 

d. grouping the vertices of the wireframe model 
into regions having a characteristic 
indicating abnormal structure; 

e. analyzing the regions having abnormal 
structure to identify regions having a high 
degree of abnormality; and 

f . rendering the wireframe model in an 
interactive three-dimensional display to 
indicate the regions having abnormal 
structure . 



y 14 . The method of Claim 13 wherein the segmentation 

flj step comprises the steps of : 

J: a. selecting a voxel near a surface of 

iU interest ; 

b. identifying a seed voxel from the selected 
voxel ; 

c. inserting the seed voxel into an adjacency 
tree ; 

d. producing a surface patch associated with 
the seed voxel ; 

e. inserting all unexamined voxels in the 
surface patch into a seed voxel queue; 

f . removing a seed voxel from the seed voxel 
queue ; and 
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g. repeating step 14. c through step 14 . f until 
the seed voxel queue is empty. 



15. The method of Claim 13 wherein the segmentation 
step, the wireframe generating step, the grouping 
step, and the analyzing step are performed 
remotely from the volume formation step. 

16. A computer- implemented method for interactively 
displaying a three-dimensional rendering of a 
structure having a lumen comprising the steps of: 

a. forming a three-dimensional volume of data 
from a series of two-dimensional images 
representing at least one physical property 
associated with the three-dimensional 
structure; 

b. segmenting a region of interest from the 
volume of data based on a selected value of 
the physical property representing the 
region of interest; 

c. creating a surface of the region of 
interest; 

d. generating a wireframe model of the surface; 

e. deforming the surface to more closely 
conform to the region of interest; and 

f . rendering the wireframe model in an 
interact ive three - dimens ional di spl ay . 

17. The method of Claim 16 wherein the segmentation 
step, the wireframe generating step, and the 
refining step are performed remotely from the 
volume formation step. 
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18. A computer- implemented method for interactively 
displaying a three-dimensional rendering of a 
structure having a lumen comprising the steps of: 

a. forming a three-dimensional volume of data 
from a series of two-dimensional images 
representing at least one physical property 
associated with the three-dimensional 
structure ; 

b. segmenting a region of interest from the 
volume of data based on a selected value of 
the physical property representing the 
region of interest; 

c. generating a wireframe model of the surface; 

d. refining the wireframe model of the 
segmented region of interest to more 
accurately represent the region of interest; 
and 

e. rendering the wireframe model in an 
interactive three-dimensional display. 

19. The method of Claim 18 wherein the segmentation 
step approximates the region of interest and the 
refinement step adjusts the coordinate positions 
of the vertices along a normal vector associated 
with each vertex. 

20. The method of Claim 19 wherein the refinement 
step adjusts the coordinate positions of the 
vertices along a normal vector associated with 
each vertex at a rate that is a function of a 
surface curvature and the confidence level of a 
boundary estimate at the coordinate position. 
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21. The method of Claim 19 wherein the refinement 
step adjusts the coordinate positions of the 
vertices along a normal vector associated with 
each vertex by determining an inner wall position 
and an outer wall position from a first and a 
second degree derivative associated with the 
coordinate positions of the vertices. 



22 . The method of Claim 18 wherein the segmentation 
step, the wireframe generating step, and the 
refining step are performed remotely from the 
volume formation step. 

% 23* A method for guiding biopsy of a tissue by 

S3 accessing the tissue via a three-dimensional 

!|i structure having a lumen comprising the steps of: 

Cm a. forming a three-dimensional volume of data 

iTi from a series of two-dimensional images 

representing at least one physical property 
St associated with the three-dimensional 

^ structure and at least one physical property 

q associated with the tissue; 

^ b. segmenting a first region of interest from 

the volume of data based on a selected value 
of the physical property associated with the 
t hr e e - dimens i ona 1 s t rue t ur e ; 

c. creating a first surface of the first region 
of interest ; 

d. generating a first wireframe model of the 
surface of the first region of interest; 

e. segmenting a second region of interest from 
the volume of data based on a selected value 
of the physical property associated with the 
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tissue; 

f . creating a second surface of the second 
region of interest; 

g. generating a second wireframe model of the 
isosurface of the second region of interest; 

h. rendering the first and second wireframe 
models in an interactive three-dimensional 
environment ; 

i . inserting a simulated biopsy needle within 
the lumen of the three-dimensional 
structure ; 

j . determining the position of the simulated 
biopsy needle within the three-dimensional 
structure ; and 

k. displaying the rendered, interactive three- 
dimensional display from a perspective 
indicating the position of the biopsy needle 
within the three-dimensional structure. 



24. The method of Claim 23 comprising the step of 
indicating the position of the simulated biopsy 
needle relative to the position of the tissue. 

25. The method of Claim 24 wherein the indicating 
step comprises the steps of: 

a. generating a zone around the simulated 
tissue; and 

b. issuing an indicator when the tip of the 
simulated biopsy needle is within the zone. 

26. The method of Claim 24 wherein the indicating 
step comprises the step of determining a distance 
and a trajectory from the simulated biopsy needle 
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to the simulated tissue. 

The method of Claim 13, 16, or 18 comprising an 
image acquisition step wherein a series of 
two-dimensional images representing at least one 
physical property associated with the structure 
is acquired, wherein the image acquisition step 
comprises a patient preparation step. 

28. The method of Claim 27 wherein the patient 
preparation step comprises the step of 
administering to a patient a nonionic intravenous 
iodinated contrast bolus. 

29. The method of Claim 27 wherein the patient 
preparation step comprises the step of 
administering to a patient an oral contrast agent 
for opacifying stool and water. 

30. A system for interactively displaying three 
dimensional structures comprising: 

a. volume formation means for forming a 
three-dimensional volume of data from a 
series of two-dimensional images 
representing at least one physical property 
associated with a three dimensional body; 

b. modeling means for producing a wireframe 
model of a selected region of interest, the 
wireframe model comprising a plurality of 
vertices, each vertex having a coordinate 
position; 

c. refinement means for refining the wireframe 
model by adjusting the coordinate positions 
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of the vertices to more accurately represent 
the region of interest ; and 
d. rendering means for rendering the wireframe 
model in an interactive three-dimensional 
display, therein producing a virtual three- 
dimensional environment . 

The system of Claim 30 wherein the refinement 
means adjusts the coordinate positions of 
vertices of the wireframe model along a normal 
vector associated with each vertex at a rate that 
is a function of a surface curvature at the 
coordinate position. 

The system of Claim 31 wherein the refinement 
means adjusts the coordinate positions of the 
vertices along a normal vector associated with 
each vertex by determining an inner wall position 
and an outer wall position an analysis of 
intensity profiles of voxel values along the 
normal vector. 

The system of Claim 3 0 wherein the modeling 
means, and the refining means are located 
remotely from the volume formation means. 

A system for interactively displaying three 

dimensional structures comprising: 

a. volume formation means for forming a 

three-dimensional volume of data from a 
series of two-dimensional images 
representing at least one physical property 
associated with a three dimensional body; 
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b. wireframe producing means for producing a 
wireframe model of a selected region of 
interest of the three-dimensional volume; 

c. deforming means for deforming the wireframe 
model to more accurately represent the 
region of interest; and 

d. rendering means for rendering the wireframe 
model in an interactive three-dimensional 
display, therein producing a virtual three- 
dimensional environment . 



35. A system for interactively displaying a three- 
dimensional rendering of a structure having a 
lumen and indicating regions having abnormal 
structure comprising : 

a. volume formation means for forming a three- 
dimensional volume of data from a series of 
two-dimensional images representing at least 
one physical property associated with the 
three-dimensional structure ; 

b. wireframe generation means for generating a 
wireframe model of a selected region of 
interest ; 

c. vertex grouping means for grouping the 
vertices of the wireframe model into regions 
having abnormal structure; 

d. analyzing means for analyzing the regions 
having abnormal structure; and 

e. rendering means for rendering the wireframe 
model in an interactive three-dimensional 
display to indicate the regions having 
abnormal structure . 
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36. The system of Claim 3 5 wherein the analyzing 
means comprises means for evaluating two 
principal curvatures associated with each point 
in an elliptical region. 

37. The system of Claim 36 wherein the analyzing 
means comprises a means for identifying ridges, 

38. The system of claim 37 comprising wireframe 
deforming means for deforming the wireframe model 
to more accurately represent the region of 
interest . 



39. A computer- implemented method for interactively 
displaying a three-dimensional rendering of a 
structure having a lumen and indicating regions 
having abnormal structure comprising the steps 
of : 

a. forming a three-dimensional volume of data 
from a series of two-dimensional images 
representing at least one physical property 
associated with the three-dimensional 
structure; 

b. generating a wireframe model of a selected 
region of interest, the wireframe model 
comprising a plurality of vertices, each 
vertex having a coordinate position; 

c. grouping the vertices of the wireframe model 
into regions having a characteristic 
indicating abnormal structure; and 

d. analyzing the regions having abnormal 
structure to identify regions having a high 
degree of abnormality. 
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40, The method of claim 39 comprising the steps of 
rendering the wireframe model in an interactive 
three-dimensional display to indicate the regions 
having abnormal structure* 



41. A computer- implemented method for interactively 
displaying a three-dimensional rendering of a 
structure having a lumen comprising the steps of: 

a. forming a three-dimensional volume of data 
from a series of two-dimensional images 
representing at least one physical property 
associated with the three-dimensional 
structure; 

b. generating a wireframe model of a selected 
region of interest; 

e . deforming the wireframe model to more 
accurately represent the region of interest; 
and 

f . rendering the wireframe model in an 
interactive three-dimensional display. 

42. A computer- implemented method for interactively 
displaying a three-dimensional rendering of a 
structure having a lumen comprising the steps of: 

a. forming a three-dimensional volume of data 
from a series of two-dimensional images 
representing at least one physical property 
associated with the three-dimensional 
structure; 

b. generating a wireframe model of a selected 
area of interest; 

c. refining the wireframe model of the selected 
region of interest to more accurately 
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represent the region of interest; and 
d. rendering the wireframe model in an 

interactive three-dimensional display. 

43. The method of Claim 42 wherein the refinement 
step adjusts coordinate positions of vertices of 
the wireframe model along a normal vector 
associated with each vertex at a rate that is a 
function of a surface curvature at the coordinate 
position. 

44. The method of Claim 43 wherein the refinement 
step adjusts the coordinate positions of the 
vertices along a normal vector associated with 
each vertex by determining an inner wall position 
and an outer wall position from a first and a 
second degree derivative associated with the 
coordinate positions of the vertices. 

45. A method for guiding biopsy of a tissue by 
accessing the tissue via a three-dimensional 
structure having a lumen comprising the steps of: 

a. forming a three-dimensional volume of data 
from a series of two-dimensional imkges 
representing at least one physical property 
associated with the three-dimensional 
structure and at least one physical property 
associated with the tissue; 

b. generating a wireframe model of a selected 
region of interest; 

c. rendering the wireframe model in an 
interactive three-dimensional display; 

d. inserting a simulated biopsy needle within 
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the lumen of the three-dimensional 
structure ; 

determining the position of the simulated 
biopsy needle within the three-dimensional 
structure ; and 

displaying the rendered, interactive three- 
dimensional display from a perspective 
indicating the position of the simulated 
biopsy needle within the three-dimensional 
structure . 

46. The method of Claim 45 comprising the step of 
indicating the position of the simulated biopsy 
needle relative to the position of the tissue. 

47. The method of Claim 46 wherein the indicating 
step comprises the steps of: 

a. generating a zone around the tissue; and 

b. issuing an indicator when a tip of the 
biopsy needle is within the zone. 

48. The method of Claim 46 wherein the indicating 
step comprises the step of determining a distance 
and a trajectory from the simulated biopsy needle 
to the tissue. 

49. The method of Claim 39, 41, 42, or 45 comprising 
an image acquisition step wherein a series of 
two-dimensional images representing at least one 
physical property associated with the structure 
is acquired, wherein the image acquisition step 
comprises a patient preparation step. 
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50. The method of Claim 49 wherein the patient 
preparation step comprises the step of 
administering to a patient a nonionic intravenous 
iodinated contrast bolus . 

51. The method of Claim 4 9 wherein the patient 
preparation step comprises the step of 
administering to a patient an oral contrast agent 
for opacifying stool and water. 
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